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ABSTRACT 

It  is  clear  that  the  way  the  military  prepares  for  combat  is  rapidly  changing.  Doctrine  and  mission  objectives 
are  changing  at  a  faster  rate  than  ever  before.  The  operational  tempo  is  faster  than  it  has  been  in  the  past.  Yet 
leadership  demands  a  prepared  military  force  -  no  performance  drop-off  is  acceptable.  In  fact,  we  will  need 
to  discover  how  to  train  in  novel  ways,  in  novel  places,  on  novel  tasks,  to  meet  the  requirements.  While 
technology  is  not  the  whole  answer  to  the  puzzle,  it  must  play  a  part.  Determining  how  simulation  and  gaming 
technologies  can  be  brought  to  bear  on  readiness  will  be  key.  Measures  of  effectiveness  that  equate  to 
readiness  are  also  needed.  Much  of  what  is  done  today  is  subjective  in  nature.  There  needs  to  be  an  improved 
mix  of  subjective  and  objective  measures  that  can  be  used  to  “roll  up’’  readiness  from  the  individual  on  up  to 
an  entire  force.  Identifying  opportunities  for  simulation  and  gaming  that  really  improve  individual  and  team 
performance  and  that  are  deployable  so  that  they  can  be  used  in  theatre  are  essential  to  success. 

We  have  been  working  with  the  United  States  Marine  Corps  at  Twentynine  Palms,  California  this  past  year  on 
the  beginning  of  a  research  program  designed  to  (a)  determine  a  baseline  measure  of  how  effective  current 
training  methods  are  towards  readiness  standards,  (b)  document  the  training  measures  and  methods  of  team 
training  used  at  29  Palms,  and  (c)  identify  how  simulation  and  gaming  technologies  can  be  used  to  enhance 
current  training  methods,  and  how  these  technologies  should  be  integrated  with  conventional  training.  The 
paper  will  document  our  progress  to  date  and  will  forecast  what  is  happening  next  towards  this  important 
research  goal  that  will  have  broad  applicability  beyond  the  Marine  Corps  to  all  NATO  military  training. 


1.0  INTRODUCTION 

The  combined-arms  operations  are  one  of  the  core  competencies  in  the  US  Marine  Corps  (USMC)  -  they 
represent  integrated  organizations  of  air,  ground,  and  logistic  forces  under  a  single  commander.  The 
Combined  Arms  Training  (CAT)  is  a  part  of  the  Mojave  Viper  training  program  at  the  Marine  Cotps  Air-to- 
Ground  Combat  Centre  (MCAGCC)  in  Twentynine  Palms,  California.  It  provides  ‘graduate-level’  combined- 
arms  training  for  all  Marines  and  it  is  structured  as  a  series  of  live-fire  ground  force  training  courses  from 
squad  and  platoon  to  battalion  level.  The  CAT  training  is  followed  by  another  block  of  training  -  the  Urban 
Warfare  Training  (UWT)  that  completes  a  unit’s  training  in  the  Mojave  Viper  program. 
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Figure  1:  Scenes  from  two  Different  Training  Courses  in  Combined  Arms  Training. 


During  past  year  and  a  half  we  have  been  actively  working  with  USMC  personnel  from  Twentynine  Palms. 
This  collaboration  was  a  part  of  the  Virtual  Technologies  and  Environments  (VIRTE)  program  sponsored  by 
Office  of  Naval  Research  (ONR),  and  was  focused  on  different  aspects  of  combined  arms  training  organized 
for  USMC  units.  The  major  thrust  of  our  collaboration  was  with  two  USMC  groups:  Tactical  Training 
Exercise  Control  Group  (TTECG),  and  the  Battle  Simulation  Center.  Under  the  VIRTE  program  umbrella  we 
engaged  in  two  major  research  activities:  first,  we  worked  on  determining  the  baseline  parameters  of  training 
provided  in  CAT,  and  second,  we  engaged  on  several  transfer  studies  focused  on  the  effectiveness  of  training 
with  virtual  training  simulations  used  in  units’  preparations  for  the  CAT.  Each  study  involved  working 
actively  with  two  USMC  battalions.  We  collected  invaluable  information  that  both  supported  our  research 
efforts  and  helped  further  our  understanding  of  this  domain.  While  we  are  still  in  the  process  of  analyzing  the 
data  collected,  we  are  able  to  extract  the  results  of  qualitative  analysis  and  present  them  in  this  paper. 

We  first  provide  an  overview  of  combined  arms  training  and  give  the  rationale  as  to  why  specific  and  focused 
types  of  training  are  needed  prior  to  unit’s  coming  to  Twentynine  Palms  and  participation  in  Mojave  Viper. 
The  specifics  of  our  approach  in  conducting  users  studies  are  also  detailed  as  possible  guidance  to  readers  who 
are  planning  to  work  with  domain  users  and  practitioners.  We  detail  both  studies,  the  baseline  study  and  the 
study  focused  on  evaluation  of  training  effectiveness  with  virtual  training  simulations,  and  provide  insights 
and  lessons  learned  throughout  this  work.  Finally,  we  discuss  our  findings  and  conclude  with  an  overview  of 
research  areas  that  we  plan  to  focus  in  the  future  as  a  result  of  the  work  in  these  studies. 


2.0  COMBINED  ARMS  TRAINING 

2.1  Crawl-Walk-Run  Approach 

While  attending  the  one  month  long  training  in  Twentynine  Palms,  each  unit  goes  through  a  series  of 
classroom  events  (‘craw’  and  ‘walk’  stages  in  the  training  process),  and  they  complete  their  training  in  the 
field  with  selected  live  fire  exercises  (a  graduate  level  -  ‘run’  stage).  The  support  for  these  classes  and 
exercises  in  the  field  is  enabled  by  a  number  of  professionally  trained  and  highly  experienced  TTECG 
instructors  (called  ‘coyotes’).  They  ensure  everyone's  safety,  and  they  control,  monitor,  evaluate,  teach,  train, 
coach  and  direct  Marines  during  the  execution  of  their  training  missions  in  Mojave  Viper. 
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2.2  Specifics  of  the  Training  Environment 

Being  that  the  number  of  hours  and  days  that  can  be  spent  in  live  fire  courses  is  limited,  each  unit  as  well  as 
TTECG  instructors  need  to  make  sure  that  this  small  window  of  opportunity  is  utilized  most  effectively  and 
with  the  best  possible  training  results  achieved.  There  is  a  strong  demand  to  make  the  best  use  of  resources 
available  to  the  unit  in  Mojave  Viper,  which  is  emphasized  by  the  fact  that  most  of  these  resources  and 
opportunities  are  not  available  in  a  units’  home  base.  They  consist  of  the  following: 

•  use  of  unique,  specially  designated  terrain  of  the  appropriate  size  to  enable  training  in  each  course 
(exercise), 

•  availability  of  professional  expertise  provided  by  highly  trained  and  experienced  TTECG  instructors, 

•  use  of  costly  live  ammunition,  ordinance  and  materials, 

•  use  of  air  assets  not  usually  available  to  the  units  in  their  home  bases, 

•  use  of  complex  logistical  support, 

•  use  of  specially  designed  training  environments  provided  for  each  course:  mock-up  villages,  trenches, 
mock-up  mine  fields,  remotely  controlled  pop-up  targets  and  smoke  signals, 

•  training  under  challenging  combat  conditions:  challenging  and  disorienting  terrain,  and  excessive 
weather  conditions  (strong  winds  and  sand  storms,  yearly  temperatures  ranging  from  below  freezing 
up  to  over  120  degrees  F). 

Due  to  prohibitive  costs,  it  may  not  be  possible  to  have  units  spend  more  time  in  live  fire  exercises,  even 
though  live  fire  training  may  lead  to  better  results.  If  the  level  of  a  unit’s  preparedness  is  not  close  to  the  ‘run’ 
stage  (near  graduate  level),  the  resources  made  available  to  the  unit  in  live  fire  will  not  be  used  most 
effectively  because  the  foundation  material  has  net  yet  been  mastered.  In  such  a  situation  it  becomes 
imperative  to  invest  additional  effort  in  the  units’  training  prior  to  their  coming  to  Twentynine  Palms, 
maximizing  their  chances  for  greater  training  success  in  the  field.  We  also  emphasize  that  within  this  domain, 
no  training  system  or  set  of  training  systems  is  a  suitable  replacement  for  the  physical  nature  of  a  live  fire 
exercise  and  the  complexity  of  that  environment.  Suggesting  that  virtual  training  systems  are  possible 
replacements  for  an  entire  training  cycle  is  counterproductive  and  does  not  serve  the  community  well. 
However,  it  is  highly  appropriate  to  use  virtual  training  systems  as  a  supplement  to  regular  training  or  as  tools 
to  replace  selected  parts  of  that  training,  provided  that  the  final  goal  of  improved  performance  in  the  live  fire 
exercise  is  achieved. 

3.0  USER  STUDIES 

3.1  Our  Approach 

Our  past  work  in  collaborative  projects  that  involved  a  number  of  partners  is  extensive  and  over  time  we  have 
distilled  several  themes  that  we  see  as  crucial  to  the  success  of  these  types  of  activities.  This  approach  enables 
us  to  draw  valuable  insights  from  our  data  and  observations  where  our  conclusions  are  useful  across  multiple 
domains  and  user  populations. 

The  themes  that  guide  our  work  are: 

•  Domain  (expert/end)  user  populations:  In  studies  that  involve  specific  user  populations,  we  work 
predominantly  with  domain  (expert/end)  users  as  our  source  of  data  and  information  —  in  this  case 
military  personnel  of  all  services.  We  typically  do  not  use  surrogate  or  “representative”  user 
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populations  in  studies.  This  enables  us  to  attain  a  high  level  of  relevance  and  to  address  the  very  needs 
exercised  by  these  end  users. 

•  Extensive  subject  matter  expert  consultation:  In  each  project  activity  we  consult  heavily  with  a 
large  group  of  subject  matter  experts.  The  majority  of  our  students  are  military  officers  and  some  of 
our  faculty  have  had  a  military  career  prior  to  their  coming  to  NPS.  This  helps  us  quickly  and  clearly 
identify  the  problem  space,  connect  ourselves  with  the  relevant  knowledgebase  in  the  domain  and 
address  specific  issues  central  to  our  study. 

•  Domain  practitioners  as  project  partners:  This  is  probably  the  most  important  and  also  unique 
element  of  our  approach.  When  working  with  domain  practitioners  and  subject  matter  experts,  we 
encourage  and  accept  them  in  the  role  of  project  partners  rather  then  merely  occasional  expert 
advisers  whenever  possible.  This  approach  allows  for  much  tighter  collaboration  and  enables  a 
mechanism  for  ensuring  high  relevance  of  our  work  and  results.  Additionally,  a  collaboration  framed 
like  this  provides  for  direct  integration  of  project  results  into  the  community  of  training  providers  and 
end  users  because  they  have  taken  some  level  of  ownership  of  the  project  and  process.  This  is 
extremely  important  to  motivate  the  user  population  to  adopt  the  final  product.  The  researchers  in  the 
project  also  have  the  unique  opportunity  to  benefit  from  the  rich  experience  and  expertise  that  the 
practitioners  have  acquired  through  their  professional  jobs,  and  this  exchange  of  expertise  may  have 
been  missed  or  it  would  be  accomplished  to  a  much  lesser  extent  if  the  work  was  not  organized  in  a 
partner  relationship. 

•  Two  tier  approach:  When  engaging  domain  users  it  is  important  to  get  an  acceptance  (the  ‘buy  in’) 
on  two  levels.  One  level  is  represented  with  the  leadership  of  the  community  that  we  work  with  -  they 
need  to  have  a  clear  understanding  of  the  research  goals,  and  be  assured  that  the  resources  they  invest 
in  collaboration  will  be  well  spent.  The  other  level  of  user  community  are  the  end  users  themselves  - 
they  need  to  know  if  and  how  their  lives  and  work  will  be  augmented  with  the  use  of  systems  and 
approaches  that  we  suggest.  They  are  more  likely  to  reject  solutions  if  those  were  imposed  on  them, 
and  which  they  did  not  have  a  chance  to  fully  accept  as  theirs.  Some  elements  of  grassroot  approach 
may  be  very  effective  in  assuring  the  support  of  end  users,  yet  the  support  of  leadership  will  still  be 
needed  if  the  goal  is  to  work  with  large  number  of  end  users  and  do  it  in  an  organized  and  well 
structured  way  that  clearly  supports  unit’s  training  needs. 

•  Extensive  and  long  term  collaboration:  In  order  to  create  high  level  of  trust  essential  for  a  fruitful 
collaboration,  and  to  ensure  that  the  domain  is  well  understood,  one  needs  to  spend  considerable  time 
with  domain  users  in  their  work  environment.  We  are  unfortunately  the  frequent  witnesses  of 
situations  when  researchers  spend  insufficient  amount  of  time  observing  and  working  with  domain 
users,  and  as  a  result  they  remain  on  a  superficial  level  of  understanding  of  the  domain.  Very  short 
and  infrequent  visits  are  almost  sure  signs  that  the  results  will  be  highly  limited  and  in  some  cases 
even  misleading.  The  phenomena  and  situations  that  are  studied  may  look  ‘easy’  and  manageable 
after  two  hours  of  observation,  but  the  real  complexity  may  become  evident  only  after  two  days  of 
thorough  analysis,  interviews  and  observation.  Additionally,  a  long  term  dedication  to  collaboration  is 
another  important  aspect  that  leads  towards  greater  success  -  if  domain  users  know  that  the  work  does 
not  happen  in  ‘one  instalment’  only,  but  that  the  appropriate  follow  ups  will  be  put  in  place,  they  are 
more  likely  to  invest  more  of  their  work  energy  and  have  a  higher  level  of  appreciation  for  the 
collaboration. 

We  firmly  believe  that  these  themes,  along  with  the  nature  of  our  institution  as  an  educational  and  research 
organization,  as  well  as  the  access  that  we  have  to  the  fleet  as  a  government  institution,  provide  significant 
advantages  when  addressing  training  and  operational  needs  of  the  fleet  in  our  research. 
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3.2  Baseline  Study 

3.2.1  Objectives  and  Goals 

The  baseline  study  was  designed  to  benchmark  the  current  training  situation,  procedures,  metrics  and 
evaluation  of  the  overall  preparedness  of  multiple  units  for  collective  training.  The  training  situation  that  was 
used  to  acquire  those  insights  and  to  evaluate  a  units’  performance  was  Revised  Combined  Arms  Exercise 
(RCAX)  that  later  on  was  transformed  into  Mojave  Viper.  The  results  of  this  study  help  us  describe  a  baseline 
of  today's  training  and  to  establish  a  set  of  benchmarks  to  be  used  for  the  future  training  processes  and 
products.  The  study  also  provided  constructive  guidance  about  the  ways  in  which  training  simulations  could 
assist  in  future  training. 

This  study  involved  no  direct  intervention  on  our  part  and  was  primarily  focused  on  collecting  subjective  and 
objective  data,  and  our  observations.  The  data  were  collected  in  a  manner  that  allows  us  not  only  to  acquire 
the  final  results  (summative  analysis)  but  also  to  have  a  good  understanding  of  the  process  that  units  went 
through  while  preparing  for  different  courses  in  the  live  fire  exercise  (formative  analysis).  We  organized  data 
collection  through  three  sets  of  questionnaires.  The  first  was  administered  two  to  three  months  prior  to  the 
unit’s  participation  in  RCAX.  The  second  was  administered  just  before  the  unit  went  to  RCAX,  and  the  third 
was  administered  immediately  after  RCAX.  Each  questionnaire  took  about  20  minutes  to  complete.  We  also 
conducted  a  small  number  of  in-depth  interviews  with  trainees  (Marines),  unit  commanding  officers,  TTECG 
personnel,  and  we  recorded  observations  and  evaluations  that  TTECG  instructors  made  for  each  course  in  the 
live  fire  exercise. 

3.2.2  Lessons  Learned 

Team  performance  measurements:  When  attempting  to  evaluate  performance  of  a  group  of  individuals, 
especially  if  their  performance  consists  of  a  set  of  actions  that  do  not  have  clear-cut  quantitative  results 
assigned  to  them,  one  is  confronted  with  several  interesting  problems.  One  is  a  tendency  to  aggregate  the 
results  to  provide  a  unified  measure  of  group  performance.  If  we  say  that  a  certain  unit’s  performance  was 
‘average’,  we  have  very  little  if  no  understanding  that  would  help  us  compare  that  unit  with  others.  Two 
‘average’  units  may  actually  be  quite  different.  In  this  situation  we  chose  not  to  aggregate  results  but  rather 
remain  at  the  last  level  that  provides  information  useful  for  further  processing  and  consideration.  The  second 
problem  is  associated  with  using  quantitative  measures  outside  of  the  full  context  in  which  they  were 
obtained.  An  example  may  be  presented  with  measuring  performance  of  two  teams  that  engage  in  clearing  the 
same  set  of  rooms.  Imagine  that  team  A  finishes  in  10  minutes  and  team  B  in  25  minutes.  Not  having  further 
details  about  their  performance  one  could  conclude  that  team  A  was  more  effective.  Had  we  known  that,  for 
example,  team  A  lost  60%  of  their  personnel,  and  team  B  completed  with  all  team  members  alive,  we  would 
draw  very  different  conclusion.  This  line  of  consideration  warns  us  to  be  very  careful  when  dealing  with  any 
measure  in  isolation  from  other  values  collected  in  the  same  context,  as  that  may  provide  misleading 
conclusions.  Also,  quantitative  measures  are  best  understood  if  supplemented  with  qualitative  measures. 
Together  they  provide  a  full  picture  and  enable  a  thorough  understanding  of  the  domain.  The  same 
consideration  holds  for  the  subjective  and  objective  data  -  we  need  both  to  be  able  to  have  full  insights  into 
the  domain. 

Performance  trends:  Our  observations  suggest  that  the  existence  and  number  of  performance  trends 
exhibited  by  the  unit,  as  well  as  unique  items  for  a  particular  unit,  may  represent  a  preferable  tool  when 
evaluating  the  level  of  preparedness  for  a  given  unit  in  live  fire  training.  While  no  unit  is  identical  to  any 
other,  there  are  remarkable  similarities  in  their  performance.  Beside  very  good  performance  that  a  unit  may 
demonstrate,  they  tend  to  repeat  some  trends  that  are  qualified  as  negative.  As  an  illustration,  loading  weapons 
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while  kneeling  or  standing  while  exposed  is  a  negative  trend.  Our  main  sources  of  objective  measurements 
were  evaluations  collected  from  TTECG  instructors.  When  evaluating  a  unit’s  performance  TTECG 
instructors  will  mention  outstanding  (good)  performances,  but  they  will  primarily  focus  on  things  that  need  to 
be  improved.  Therefore,  their  evaluation  will  have  a  plethora  of  information  related  to  negative  trends  and 
limited  information  on  things  that  went  well.  This  is  understandable  because  in  the  limited  time  allotted  to 
debrief  the  unit,  the  instructors  need  to  provide  the  best  guidance  on  where  to  focus  further  training  efforts, 
and  how  to  improve  the  overall  level  of  readiness  of  that  unit.  The  negative  performance  trends  are  a  clear 
indication  of  the  areas  where  the  supplementary  use  of  virtual  training  systems  focused  specifically  on  those 
skills  may  be  considered.  They  also  may  point  to  new  features,  new  types  of  training,  or  even  new  systems 
and  lines  of  research  that  need  to  be  introduced  to  help  address  those  negative  performance  trends.  Closely 
associated  with  this  is  a  need  for  more  systematic  identification  and  documenting  of  negative  performance 
trends. 

Video  games:  according  to  our  preliminary  statistics  more  than  60%  of  Marines  play  video  games  in  their  free 
time.  Given  the  fact  that  they  do  so  on  daily  basis  and  for  several  hours  each  day,  it  represents  potential  that  is 
waiting  to  be  tapped.  Since  video  games  are  associated  with  something  that  people  enjoy  doing  and  are  very 
passionate  about,  bringing  this  energy  and  enthusiasm  into  the  work  environment  through  the  use  of  such 
systems  and  tools  can  only  benefit  that  work  environment.  Additionally,  the  same  people  will  already  be 
experts  in  basic  skills  that  one  would  need  to  spend  time  teaching  them  even  before  the  training  starts 
(navigation  and  wayfinding  in  3D  worlds,  object  manipulation,  remote  collaboration,  and  similar). 

Constructive  integration  of  a  set  of  skills:  People  are  capable  of  mastering  individual  skills  in  a  ‘disjoint’ 
fashion,  but  they  experience  new  problems  when  the  same  skills  need  to  be  combined  in  a  set  of  connected 
events  where  timing  and  priorities  of  several  tasks  need  to  be  decided,  and  when  a  set  of  tasks  may  need  to  be 
concurrently  performed.  Our  hypothesis  is  that  if  the  training  is  presented  in  ‘chunks’,  without  making  critical 
and  causal  connection  between  those  ‘chunks’,  the  likelihood  of  experiencing  problems  when  integrating 
learned  knowledge  and  skills  is  higher.  The  design  of  new  systems  and  training  approaches  should  take  this 
into  consideration  and  ensure  that  necessary  elements  of  skills  integration  are  included  whenever  possible. 

Learning  of  evaluators:  While  observing  TTECG  instructors  in  their  job,  we  became  fully  aware  of  the 
uniqueness  of  their  position  and  special  learning  process  that  they  were  going  through.  Being  that  they  are  not 
part  of  the  units  that  come  to  Mojave  Viper,  their  position  is  neutral  which  allows  for  more  objective 
observation  and  evaluation,  and  consequently  they  are  able  to  notice  and  comment,  on  mistakes  much  easier. 
The  learning  enabled  in  this  context  carries  extraordinary  value,  and  it  can  be  regarded  as  a  great  learning 
opportunity  for  all  future  unit  leaders  and  commanders.  We  were  told  that  some  units  do  send  their 
commanding  officers  to  observe  live  fire  exercises  prior  to  the  unit  coming  to  Twentynine  Palms,  but  this  is 
not  done  on  a  systematic  and  organized  basis  (it  is  also  not  required).  There  is  only  anecdotal  evidence  about 
those  units  exhibiting  a  smaller  number  of  negative  performance  trends,  but  we  did  not  have  the  opportunity 
to  look  into  this  and  confirm  it  in  our  studies.  It  would  be  interesting  to  see  what  kind  of  improvement  in  a 
unit’s  performance  could  be  registered  if  commanding  officers  were  required  to  spend  some  time  with 
TTECG  instructors,  helping  (shadowing)  current  instructors,  and  bringing  that  knowledge  back  to  the  unit. 
While  one  can  argue  whether  this  opportunity  can  be  afforded  to  all  units  or  not,  it  is  obvious  that  similar 
learning  experience  can  be  still  provided  to  everyone  by  presenting  them  a  set  of  well  prepared  short  video 
clips  (those  video  clips  can  be  made  using  the  same  virtual  training  systems  that  the  trainees  will  use  in  their 
training).  In  this  situation  the  trainees  would  be  asked  to  observe,  evaluate  and  comment  on  performances  they 
just  saw.  Being  that  they  are  still  the  neutral  observers  (it  is  not  their  own  performance  they  would  be 
judging),  they  would  have  that  relative  distance  and  reduced  level  of  pressure  that  is  adequate  for  the 
beginning  of  their  training.  Our  hypothesis  is  that  the  same  trainees  would  be  less  inclined  and  less  likely  to 


20-6 


RTO-MP-HFM-136 


Combined  Arms  Training:  Measures  and  Methods  for  a  Changing  World 


repeat  the  same  mistakes  that  they  just  observed,  evaluated  and  (perhaps  even  harshly)  critiqued.  In  this 
context  two  different  technologies  are  seen  as  supporting  each  other  and  enabling  more  effective  training 
overall.  We  would  like  to  note  here  that,  as  the  training  progresses  and  the  trainees’  skills  mature,  they  should 
be  put  in  situations  where  they  need  to  evaluate  and  provide  constructive  critique  for  their  own  performances. 

Longitudinal  studies:  We  were  eager  to  engage  in  longitudinal  studies  over  multiple  years,  and  to  follow  up 
the  development  of  one  unit’s  performance  level  over  time.  However  this  was  not  possible  for  two  reasons: 
each  unit  changes  dramatically  in  the  course  of  one  year.  The  next  year  it  is  almost  a  completely  different  unit. 
Along  with  that  fact,  people  advance  in  their  careers,  and  the  position  that  they  had  last  year  is  most  likely  to 
be  different  next  year  when  they  are  essentially  just  beginning  to  learn  more  about  that  particular  (new) 
position.  Both  reasons  result  with  units  being  two  different  constellations  of  individuals  in  each  year. 

3.3  Evaluation  of  Training  Effectiveness 
3.3.1  Objectives  and  Goals 

The  goal  of  this  study  was  to  evaluate  the  initial  effectiveness  of  the  use  of  a  virtual  training  system  in  support 
of  training  organized  for  the  unit  prior  to  unit’s  participation  in  Mojave  Viper  (trainees  used  those  systems  for 
the  first  time).  We  have  collected  data  related  to  training  organized  in  support  of  the  Collective  Training 
Standards  in  convoy  operation  training  situations,  and  we  are  currently  working  on  another  study  that  involves 
training  of  Fire  Support  Teams  (FiST  teams).  The  work  on  both  studies  will  provide  a  constructive 
contribution  towards  the  program  (VIRTE)  mission:  evaluation  of  the  effectiveness  of  training  simulations, 
defining  instructional  strategy  matrix,  training  transfer,  team  metrics  and  team  performance. 

Our  hypothesis  was  that  the  strategic  use  of  virtual  training  simulations  as  a  supplement  for  regular  training, 
will  bring  positive  impact  on  unit  preparedness,  affecting  unit  skill  acquisition,  skill  transfer  and  retention 
rate,  unit  performance  and  motivation,  their  understanding  of  task  complexity,  and  a  shared  sense  of 
belonging  to  the  organized  group  of  individuals  working  together.  Currently  we  have  only  anecdotal  evidence 
given  by  the  practitioners  about  the  positive  impact  of  this  training  tool,  and  our  studies  were  designed  to  help 
us  establish  a  solid  basis  to  exercise  scientific  rigor  in  drawing  conclusions. 

Both  studies  involve  a  training  intervention  that  consisted  of  supplementary  use  of  virtual  training  systems  in 
conjunction  with  the  unit’s  regular  training.  The  systems  used  in  the  study  with  convoy  operations  were  a 
desktop  training  system  —  Virtual  Battlefield  Simulation  (VBS  [2]  by  Coalescent  Technologies  Co.,  Figure  2 
(a))  —  and  immersive  ‘wrap-around’  CAVE-like  system  —  Virtual  Combat  Convoy  Trainer  (VCCT  [3]  by 
Lockheed  Martin  and  FATS  Inc.,  Figure  2  (b)).  The  system  we  plan  to  use  in  the  second  study  with  FiST 
teams  was  designed  and  implemented  by  the  VIRTE  team.  The  core  of  this  training  environment  consists  of 
Forward  Observer  (FO)  [1]  and  Forward  Air  Controller  (FAC)  PC  training  simulations  initially  built  by 
MOVES  students  and  staff. 
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Figure  2:  (a)  Training  using  VBS  System,  and  (b)  Training  using  VCCT  System. 


3.3.2  Lessons  Learned:  Perceived  Trends  in  Using  Virtual  Training  Systems 

We  had  the  opportunity  to  spend  considerable  time  observing  the  Revised  Combined-arms  Exercise  (RCAX) 
and  Mojave  Viper  as  well  as  training  organized  with  VBS  and  VCCT  virtual  training  systems,  and  our 
preliminary  findings  confirm  what  we  initially  expected.  The  issues  of  availability  of  good  training 
simulations  cannot  be  treated  separately  from  the  issues  of  availability  of  equally  good  training  approaches 
and  methodologies.  The  two  are  tightly  and  critically  connected,  and  together  they  predict  the  success  of 
training  in  virtual  training  systems.  In  other  words,  no  variety  of  virtual  training  systems,  no  matter  how 
feature  rich  they  may  be,  will  have  a  great  impact  unless  we  ensure  that  people  use  them  in  a  well  structured 
process  and  framework  —  the  right  people  receive  the  right  training,  at  the  right  time,  for  the  right  period  of 
time,  with  well  trained  instructors,  and  with  the  right  training  methodologies  that  fully  support  the  training 
objectives  and  associated  operational  needs. 

To  illustrate  our  point  we  will  describe  the  most  common  trends  that  we  identified  concerning  the  use  of 
virtual  training  systems: 

•  Short  exposure  to  the  training  system:  The  total  time  that  the  trainees  spend  using  virtual  training 
systems  is  insufficient.  The  usual  approach  is  to  have  one  3-4  hour  long  ‘one-shot’  training  session. 
That  number  of  hours  is  not  sufficient  to  acquire  and  practice  a  complex  set  of  individual  and  team 
skills. 

•  One-time  exposure  only:  Trainees  receive  training  in  one  block  of  several  hours  only,  with  no 

possibility  (usually  no  time  in  their  schedule)  to  use  training  systems  again.  There  is  no  repeated 

exposure  to  a  particular  training  system  and  no  opportunity  to  do  proper  solidification  and  corrections 
of  their  skill-set. 

•  Timing  not  appropriate:  Trainees  receive  training  either  long  before  training  in  a  live  fire  exercise 
(or  some  other  situation  when  they  need  to  use  those  skills)  or  right  before  such  exercise.  In  the  case 
of  the  former,  the  retention  rate  of  perishable  skills  may  be  too  short  to  be  able  to  retain  the  necessary 
level  of  the  same  skills  learned  long  ago  in  training  simulations.  In  the  case  of  latter,  there  is  not 
enough  time  to  reflect  and  do  proper  improvement  and  corrections  of  skills  if  that  may  be  needed. 

•  Mastering  individual  skills  first:  Team  training  is  frequently  organized  with  no  special  care  given  to 

mastering  individual  skills  prior  to  that.  Ideally,  the  initial  skill  level  of  each  trainee  should  be 

evaluated  and  proper  individual  training  provided  before  they  engage  in  team  training.  Quite  often  it  is 
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important  to  incorporate  additional  prep-classes  to  bring  everyone’s  skill  level  to  a  higher  (necessary) 
degree,  and  maximize  chances  for  best  training  results  when  using  virtual  training  simulations. 

•  Disconnect  between  training  systems  and  (right)  users  of  those  systems:  Trainees  who  end  up 
using  virtual  training  systems  are  not  necessarily  the  Military  Occupation  Specialties  (MOSs)  that 
should  receive  that  particular  training.  It  may  well  happen  that  people  who  received  training  were  the 
ones  who  were  simply  available  at  the  time. 

•  Training  or  ‘fun’?:  Some  trainees  see  this  kind  of  training  as  'gaming'  (i.e.  fun)  and  may  not 
approach  it  with  the  right  attitude  where  they  can  learn  from  the  experience. 

•  Need  for  training  relevance:  There  may  not  be  very  tight  and  regularly  updated  connection  between 
the  training  provided  in  simulations  and  the  one  in  the  live  fire  exercise. 

•  Need  to  support  the  same  training  objectives:  Centers  providing  this  type  of  training  may  have 
insufficient  understanding  of  all  training  procedures  and  important  training  parameters  established  and 
observed  by  the  TTECG. 

•  Need  for  specially  trained  instructors:  Instructors  that  organize  training  in  training  simulations  are 
either  military  personnel  or  contractors  (a  large  majority  of  them  are  former  military  personnel). 
Typically,  neither  has  received  formal  training  to  be  an  instructor  in  this  kind  of  training  environment, 
(and  it  is  indeed  a  different  training  environment  from  what  they  are  used  to  in  a  more  traditional 
training  setup  and  for  which  they  are  well  versed.)  Therefore  appropriate  training  is  very  much 
needed.  Best  ‘guesses’  as  to  how  such  training  should  be  organized  is  not  sufficient  if  the  goal  is  to 
use  resources  most  effectively  and  have  highly  trained  military  forces. 

•  Need  for  solid  training  approaches  and  schedules:  There  is  a  lack  of  well  scripted  and  documented 
robust  training  routines  and  methodologies.  There  are  too  many  'on  the  fly'  and  'ad  hoc'  approaches 
where  each  instructor  has  his  own  individual  way  of  training. 

•  Accountability:  There  is  no  identified  accountability  for  training  results  achieved  in  these  training 
systems.  We  are  not  sure  how  this  could  be  remedied,  but  one  way  of  handling  it  would  be  to  use 
motivation  and  peer-competitions  as  an  alternative  tool  to  maximize  potential  that  such  systems 
provide. 

•  System  support  for  AAR:  The  systems  do  not  provide  good  information  for  After  Action  Reviews 
(AAR)  or  if  they  do,  it  is  not  used  in  a  systematic  way.  After  the  training  session,  unit  leaders  do 
perform  AAR  but  it  is  based  purely  on  their  observation  and  is  rarely  backed  with  the  data  recorded 
by  the  system. 

•  System  rigidity:  A  large  majority  of  systems  are  very  rigid  in  the  sense  that  they  do  not  support  user- 
specified  scenarios,  scripts,  use  of  terrains,  terrain  features,  weapon  systems,  and  objects  of  users’ 
choice. 

A  global  trend  that  we  see  in  training  community  is  using  approaches  and  systems  that  may  be  sufficient  and 
good  for  categories  of  users  that  fall  into  the  groups  of  innovators  and  early  adopters  -  those  users  do  not  need 
a  great  deal  of  convincing,  the  systems  they  use  do  not  need  to  be  perfected  and  they  can  manage  with  little  if 
no  additional  structure  added  to  training.  However,  as  these  groups  are  extremely  small  in  their  number,  the 
objective  of  training  community  must  remain  on  creating  technologies  and  approaches  that  will  work 
successfully  with  large  number  of  people  -  the  early  and  late  majority  of  users  [4],  These  groups  need  far 
more  structure,  support  and  motivation  in  training  cycle,  and  the  systems  they  use  need  to  be  adjusted  to  their 
knowledge  and  skill  levels. 
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4.0  DISCUSSION 

4.1  Augmenting  Virtual  Training  Systems  with  Lessons  From  the  Field 

Training  organized  in  a  live  fire  exercise  reflects  professional  and  corporate  knowledge  acquired  over  a  large 
number  of  years,  and  as  such  represents  an  ample  resource  when  defining  parameters  of  a  good  (ideal) 
training  environment.  The  observed  runs  on  each  training  course  offered  the  following  insights  that  could  be 
used  to  augment  current  virtual  training  systems. 

Dynamically  responsive  systems:  The  performance  of  each  unit  is  carefully  monitored  by  TTECG 
instructors.  They  first  act  as  independent  observers  and  let  the  unit  wrestle  with  the  problem.  If  the  unit  is 
capable  of  the  current  level  of  difficulty,  the  instructors  usually  add  new  elements  and  extend  training 
objectives  to  match  the  unit’s  performance.  If,  however,  the  unit  is  observed  to  experience  difficulties,  they 
receive  proper  guidance  and  coaching,  and  in  some  cases  the  overall  level  of  difficulty  of  that  training  run  can 
be  adequately  reduced  to  ensure  that  unit  has  enough  time  to  accomplish  (other)  most  important  training 
objectives.  The  current  knowledge  of  the  unit  is  judged  and  evaluated  on  the  spot  (instructors  know  nothing 
about  the  unit’s  level  of  preparedness  prior  to  unit’s  coming  to  Twentynine  Palms!),  and  both  coaching  and 
intervention  are  adjusted  to  meet  that  knowledge  and  skill  level. 

When  we  try  to  translate  this  complex  process  to  the  domain  of  virtual  training  systems,  one  can  imagine  that 
some  functionality  could  be  accomplished  by  having  an  automated  training  and  knowledge  management 
system.  This  system  would  provide  some  level  of  expert  guidance  on  the  fly.  Yet,  we  still  recognize  that  such 
highly  sophisticated  levels  of  instructor  intervention  may  well  never  be  achieved  with  automated  systems,  and 
therefore  if  our  goal  is  to  achieve  the  most  effective  training,  it  is  appropriate  to  recognize  and  distinguish  the 
tasks  and  training  interventions  that  can  be  automated  and  the  tasks  that  are  best  served  by  having  human 
expert  intervention  in  the  loop.  Enforcing  automated  support  for  parameters  that  cannot  be  effectively 
supported  is  seen  as  counterproductive.  It  may  also  give  us  a  false  sense  that  we  achieved  something  by 
‘engaging’  the  system  to  take  care  of  it,  while,  at  the  same  time,  that  may  be  quite  far  from  the  truth. 

Automated  AAR  and  continuous,  organic  system  adaptation:  There  are  two  types  of  internal  debriefings 
done  after  each  training  run  in  live  fire  training.  The  ‘coyotes-to-coyotes’  debriefing,  Figure  3  (a),  consists  of 
reports  of  all  field  instructors  given  to  the  main  instructor  who  compiles  everyone’s  comments  into  the 
information  that  will  be  presented  to  the  unit  later  on.  Ideally,  the  virtual  training  system  should  provide 
exactly  the  same  information:  both  a  summary  of  the  overall  unit’s  performance  as  well  as  information  that 
reflects  the  way  in  which  each  part  of  the  training  run  was  accomplished.  A  second  part  of  ‘coyotes’ 
debriefing  is  related  to  the  internal  affairs  and  performance  of  all  instructors  in  the  field.  This  is  an  opportunity 
for  the  group  to  discuss  and  critique  approaches  and  strategies  that  did  and  did  not  work  well,  and  to  identify 
what  changes  (if  any)  should  be  introduced  for  the  same  course  the  next  time.  As  a  result,  no  two  runs  of  the 
same  course  are  identical  -  they  grow  organically  and  at  any  given  time  reflect  the  level  of  understanding  that 
instructors  have  acquired  for  a  certain  training  situation.  We  see  this  as  a  great  example  of  “innovation  in 
action”,  an  ability  and  readiness  to  change  one’s  acting  and  parameters  of  working  environment  as  soon  as  a 
better  and  more  effective  solution  is  perceived  as  needed.  While  it  is  difficult  to  imagine  that  one  virtual 
training  system  could  have  this  level  of  flexibility  and  be  able  to  change  from  one  training  session  to  the  very 
next,  it  is  still  possible  to  introduce  some  elements  through  the  concepts  of  user-defined  scenarios  and  scripts 
that,  to  some  extent,  may  provide  a  much  needed  level  of  innovation  in  successive  training  sessions.  This 
becomes  an  imperative  if  the  same  training  system  is  to  be  used  multiple  times  for  training  the  same  group  of 
individuals. 
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Expert  guidance  for  follow  up  training:  After  each  training  run  TTECG  instructors  conduct  a  detailed 
debriefing  with  the  unit  (‘coyotes-to-unif  debriefing,  Figure  3  (b)),  where  they  report  about  different  aspects 
of  the  unit’s  performance.  At  the  end  of  that  debriefing  session  the  main  instructor  provides  an  account  and 
critique  for  the  entire  training  run.  This  debriefing  is  always  combined  with  suggestions  for  improvements  that 
can  be  readily  used  to  plan  follow-up  training  actions.  Bringing  this  to  the  domain  of  virtual  training  systems, 
the  effective  approach  would  be  to  provide  the  same  level  of  consideration  and  advice,  making  sure  that  the 
group  has  a  clear  understanding  of  what  kind  of  skills  they  need  to  improve  and,  equally  important,  advice  on 
how  to  go  about  it. 


Figure  3:  (a)  Internal  Debriefing  (‘Coyotes-to-Coyotes’),  and  (b)  Unit  Debriefing  (‘Coyotes-to-Unit’). 

Systems  and  approaches  that  train  for  the  values  highly  regarded  by  domain  users:  While  observing 
training  runs  in  Mojave  Viper,  we  identified  a  number  of  general  skills  that  are  highly  regarded  by  domain 
practitioners: 

•  be  able  to  conduct  thorough  mission  planning, 

•  have  high  awareness  of  what  each  person  needs  to  know  and  do  and  how  it  affects  others, 

•  have  rigor  in  execution  when  needed  but  also  allow  for  flexibility  when  that  is  needed, 

•  be  able  to  change  the  plan  on  the  fly, 

•  be  able  to  handle  unexpected  situations,  and 

•  be  able  to  ‘parallel  process’  and  work  on  preparing  the  next  missing  while  still  executing  the 
current  mission. 

Training  systems  and  training  approaches  that  would  provide  some  elements  of  this  would  be  offering 
considerable  contribution  towards  producing  ‘always  ready’  and  ‘always  maximally  ready’  forces. 

All  these  examples  refer  to  possible  augmentations  of  virtual  training  systems,  but  the  same  lessons  are 
equally  relevant  and  can  be  applied  to  any  training  (learning)  system. 
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ORGANIZATION 


5.0  FUTURE  WORK 

There  are  a  number  of  questions  that  we  would  like  to  address  in  this  line  of  research,  and  we  mentioned  some 
of  them  in  Sections  3.  and  4.  Here  we  list  other  interesting  research  issues: 

•  What  is  the  effectiveness  of  training  simulations  that  we  use  today  and  what  are  the  appropriate 
measures? 

•  How  well  do  we  incorporate  them  into  the  training  process? 

•  Do  we  use  the  right  technologies,  the  right  training  approaches,  at  the  right  times,  for  the  right 
audience  and  for  the  right  purposes? 

•  What  kind  of  technologies  and  training  approaches  can  be  used  to  support  and  to  enhance  team 
leadership  skills?  What  do  we  need  to  provide  to  strengthen  that  role?  [5] 

•  What  (minimal)  investments  with  different  technologies  can  have  the  greatest  impact  on  learning, 
training  and  motivation? 

•  Can  we  use  appropriate  combinations  of  different  technologies  and  leaming/training 
methodologies  so  that  they  encompass  learning  styles  exercised  by  the  majority  of  trainees? 

•  How  can  we  address  issues  of  Human  System  Integration  (HSI)  so  that  these  training  systems 
respond  to  perceived  training  needs,  and  the  results  of  that  training  are  fed  back  to  the  system 
design  (and  consequently  ‘close  the  loop’)? 

•  What  type  of  field  instrumentation  can  be  designed  to  support  HSI  concepts  and  at  the  same  time 
produce  valuable  resources  for  training? 

As  of  now,  the  research  community  does  not  have  good  answers  for  these  questions,  especially  when  we  apply 
them  to  training  situations  that  deal  with  collective  performance. 

We  see  our  future  research  efforts  continuing  down  the  line  of  working  with  domain  users  and  domain 
practitioners,  drawing  from  our  experiences  from  applied  field  research,  and  bringing  them  into  the  domain  of 
basic  research.  The  constant  intermingling  and  tight  connection  between  applied  and  basic  research  is  seen  as 
the  best  approach  to  ensure  that  the  investments  made  in  this  field  will  be  highly  relevant  and  addressed  with 
priorities  perceived  in  the  domain. 


6.0  CONCLUSION 

In  this  paper  we  provided  a  snapshot  of  our  reflections  and  up-to-date  results  from  research  that  we  conducted 
in  collaboration  with  USMC  and  supported  by  ONR.  A  complete  analysis  that  integrates  quantitative  and 
qualitative  results  gathered  so  far  is  planned  to  be  completed  this  year.  The  work  on  second  transfer  study  with 
FiST  team  training  will  be  finished  in  2007.  Together,  those  results  will  provide  much  better  understanding  of 
the  applicability  of  training  interventions  with  the  means  of  virtual  training  systems,  how  to  measure  the 
effectiveness  of  that  training,  and  help  us  define  necessary  guidelines  for  a  large  scale  and  a  long  term 
deployment  of  those  training  environments  in  military  settings. 

The  virtual  training  systems  and  training  methodologies  that  provide  ideal  support  for  combined  arms  training, 
and  for  that  matter  any  training  in  military  domain,  need  to  integrate  several  important  aspects  discussed  in 
more  details  in  this  paper.  They  need  to  harness  invaluable  experiences  from  the  field  training  and  deploy 
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most  effective  techniques  into  technology-supported  training  systems,  they  should  be  a  result  of  close 
collaborations  with  domain  users  and  practitioners,  and  the  additional  efforts  need  to  be  invested  so  that 
systems  and  training  approaches  are  designed  and  developed  specifically  for  the  large  number  of  users  (early 
and  late  majority  categories  of  users).  It  also  needs  to  become  a  matter  of  regular  practice  that  training  systems 
will  not  be  delivered  and  deployed  without  providing  a  comprehensive  guidance  on  the  best  training 
approaches  designed  specifically  for  that  system  for  intended  audience  -  a  whole  ‘package’  needs  to  be  made 
available  to  the  users  if  the  maximal  return  on  investment  is  expected  and  needed. 

As  the  military  doctrine  and  mission  objectives  are  changing  at  a  faster  rate  than  ever  before,  there  is  a 
growing  need  to  train  in  novel  ways,  in  novel  places,  under  the  novel  conditions  and  on  novel  tasks.  The 
virtual  training  simulations,  if  used  in  an  appropriate  framework,  can  be  seen  as  a  powerful  tool  in  ensuring 
that  those  capabilities  are  provided  in  the  training  domain,  and  that  the  leadership  demands  for  an  always  and 
maximally  prepared  military  force  are  fully  supported. 
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Combined  Arms  Exercise 
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Combined  Arms  Exercise 


•  Graduate  level  combined  arms  training  for 
USMC  units 

•  Special  /  unique  resources  and  training 
opportunities 

•  TTECG  instructors:  control,  monitor,  evaluate, 
coach,  teach,  train,  direct 

•  Organically  changing  over  the  time 

•  'Crawl  -  walk  -  run'  approach 
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Our  Approach 


1.  Work  with  the  domain/ expert  user  population 

2.  Extensive  subject  matter  expert  consultation 

3.  Domain  practitioners  as  project  partners  and 
sources  of  innovation 

4.  Two  tier  approach 

5.  Extensive  and  long  term  collaboration 
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Baseline  Study 


1.  Objectives  and  goals 

2.  Subjects:  2  USMC  battalions 

3.  Objective  and  subjective  data 

4.  Lessons: 

•  team  performance  measurements 

•  performance  trends 

•  video  games 

•  constructive  integration  of  a  set  of  skills 

•  learning  of  evaluators 
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Evaluation  of  Training  Effectiveness 


Perceived  Trends  in  Using  Virtual  Training  Systems: 

•  short  and  one-time  exposure  to  training  systems  only, 

•  timing  not  appropriate, 

•  training  objectives  in  virtual  and  real  may  not  be  well 
aligned, 

•  individual  skills  not  at  the  appropriate  level 

•  training  relevance 

•  instructors  may  not  have  formal  training  to  provide 
instructions  in  virtual  training  systems, 

•  solid  training  approaches  and  schedules  missing, 

•  no  effective  and  extensive  system  support  for  AAR. 
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Discussion 


1.  Need  to  evaluate  users  before  they  even  start 
training  -  meet  them  where  they  are 

2.  Dynamically  responsive  systems:  evaluation  of 
training  performance  on  the  fly 

3.  Automated  AAR  and  continuous,  organic  system 
adaptation  to  account  for  changes  in  training 
approaches  ('coyotes- to- coyotes'  debriefing) 

4.  Expert  guidance  for  follow  up  training  ('coyotes- 
to-unit'  debriefing) 
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Future  Directions 


1.  Right  technologies,  right  training  approaches,  at  the 
right  times,  for  the  right  audience  and  for  the  right 
training  purposes. 

2.  Systems  and  approaches  that  train  for  the  values  highly 
regarded  by  domain  users. 

3.  Synergy  of  different  technologies:  how  do  we  combine 
different  technologies  for  maximal  training  results? 

4.  What  is  needed  to  address  the  needs  of  majority  of 
users  (early  and  late  majority)? 

5.  How  do  we  close  the  loop  and  integrate  training 
systems  with  the  rest  of  training  cycle?  ->  Field 
instrumentation  in  support  of  HSI. 

6.  24/7  access  to  training  tools  and  systems. 
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Research  Question(s) 


Issues  of  availability  of  good  training  simulations  cannot  be 
treated  separately  from  the  issues  of  availability  of  equally 
good  training  methodologies  and  pedagogies: 

•  technology  alone  is  not  going  to  bring  (good)  results, 

•  development  of  both  needs  to  happen 
simultaneously, 

•  need  to  start  providing  a  'total  package'. 

Q:  When  we  work  with  the  training  systems,  should  we 
even  consider  doing  studies  that  do  not  employ  well 
designed  training  methodologies  and  pedagogies? 
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